Cross-Language Relevance Assessment and Task Context

An experiment on how users assess relevance in a foreign language they know well is reported. Results show that relevance assessment in a foreign language takes more time and is prone to errors compared to assessment in the reader’s first language. The results are related to task and context and an enhanced methodology for performing context-sensitive studies is reported. 1. Cross-linguality and reading 1.1 People are naturally multi-lingual For people in cultures all around the world competence in more than one language is quite common and the European cultural area is typical in that respect. Many people, especially those engaged in intellectual activities are familiar with more than one language and have some acquaintance with several. 1.2 People are good at making relevance assessments Information access systems deliver results which on a good day hold up to forty per cent relevant items. It is up to the reader to winnow out the good stuff from the bad. We know that readers are excellent at making relevance assessments for texts. Both assessment efficiency and precision are very impressive. But how we go about it we know very little about. Practice seems to improve both assessment speed, assessment precision, and assessor confidence, but what features a reader focuses on and how they are combined has not been studied in any great detail. 1.3 Linguistic competence is a continuum Languages are tools tied to tasks. For any one task, typically people have one language they prefer to perform it in. In general, while people may have working knowledge of more than one language, it is not common for people to have equal competence in many; the first language, or the school language, or the workplace language will tend to be stronger for whatever task they are engaged in. Linguistic competence is not a binary matter: people know a language to some extent, greater or lesser. What bits of competence are important in any given situation is an ongoing discussion in the field of language teaching – we will here concentrate on some aspects of reading, related to situation, task, and domain. 1.4 Assessing relevance in a strange language is hard – and important We do know that reading about strange things in strange genres takes more time than familiar genres, and that reading a language we do not know well is hard work, and something we only attempt if we believe it is worth the effort. Judging trustworthiness and usefulness of documents in a foreign language is difficult and a noticeably less reliable process than doing it in a language and cultural context we are familiar with. These starting points have immediate ramifications for the design of cross-lingual and multi-lingual information access systems. Presenting large numbers of documents to users if it is likely they will not be able to determine their usefulness is a waste at best and a trustworthiness and reliability risk at worst. 1.5 Finding out more – does language make a difference? We need more data about reading and related processes. To find out more we set up an experiment where Swedish-speaking subjects, fluent in English as determined by self-report, were presented with retrieval results both languages, and given the task of rating the results by relevance. Our hypotheses were that results for a foreign language would be more time-consuming and less competent than those for the first language. 1.6 Task-based approach to query construction and relevance assessment Generally, topicality has been the main criteria for relevance in information retrieval experiments. Our approach suggests that other criteria may come into play, especially criteria related to the task and domain at hand. For interactive information retrieval experiments, we propose to expand the original query with information about context. In this study, we want to relate the relevance assessment to a specific task situation, i.e. the subject will be given a semi-realistic situation including a domain description, and then we will investigate if the relevance assessment situation involves criteria beyond topicality.